Augmented Reality in a Shooting Game With Real Robots

This is a project I've been working on my spare time to play with my daughters. It's a game that you can play with your mobile device or a computer, being on the control of real robots.

In a nutshell this game includes:

- Video streaming from a robot mounted camera to your controlling device with some overlay layers included for rendering game features (like explosions).

- Laser gun and shields (actually they are infrared counterparts to be completely safe)

- Motors and wheels controlled by your device.

- RFID tags used for power-ups on battle field (a RFID reader is included on each robot)

- Use Wi-Fi for communicating robots and players. One robot may act as an Access Point in order to provide outdoor fun.

- This game is supposed to be played with 2 or more robots, one player controlling each one. Good for a geek party.

You will be using a web browser in your mobile device or computer in order to play this game. It should be compatible with HTML5 and should not be too old. Have been tested with latest versions of Chrome and Safari. Each robot will work as a web application server running all the game logic and will synchronize game status with each other robot.

Step 1: Robot Parts Lists

This is the parts list you need for building one robot (please note that you need at least two robots for this game):

Raspberry Pi 3 'B'
Raspberry Camera Module (v1.3 or newer)
Arduino Uno r3 (DIP)
Arduino Motor Driver Shield (tested with Adafruit Motor Driver Shield version 1)
Kit of Car Chassis + DC motors + wheels (tested with a common 4WD car chassis kit you can easily find for $ 17 in common Internet marketplaces).
PowerBank 10,000mAh with two USB ports
MFRC522 RFID module kit (13.56MHz) including some RFID cards
3D printed turret (PLA FDM). Sketch file provided in this instructable.
USB cable for powering Raspberry (Type A plug to Micro Type B plug)
USB cable for connecting Arduino to Raspberry (Type A plug to Type B plug)
USB cable for powering motors (Type A plug to TTL, or cut one end off)
Micro SD (suggested a class 4 with 16 GB)
Led IR TSAL 6200
Led any visible color 5mm (e.g. yellow)
Led RGB 5mm common anode (also works with a LED with common cathode if you change some settings later)
BC547 NPN transistor
Four TSOP 4838 IR light detectors
Swivel ball caster (if you want to build 2WD robot instead of a 4WD robot)
32 Dupont cables (male-female) plus 8 female-female.
Mini breadboard (170 points)
Resistors (1 x 10 kOhm, 2 x 220 Ohm, 1 x 100 Ohm, 2 x 47 Ohm)
8 screws + hex nuts M2.5 10mm
5 screws + hex nuts M3 10mm
3 metal angle brackets (90 degrees with each leg 3.5cm long, 2mm thick, holes about 7 mm diameter)
4 spacer rings

The software needed for both Raspberry and Arduino are discussed through this instructable.

It's possible to use some different hardware configurations with minor software settings changes. For example, it's possible to connect the Raspberry GPIO directly to the Arduino motor Driver Shield without using Arduino. You may also try some other motor driver PCB pluggable directly over Raspberry Pi (such as Adafruit DC Motor HAT for Raspberry).

Step 2: Robot Lower Plate Assembly

In this step you are going to mount the DC motors and wheels on your robot chassis plus three circuit boards (Arduino, motor driver shield and MFRC522).

I built this robot with a very common and cheap kit that includes four sets of wheels and motors, two acrylic plates, screws and some other mounting parts. Please note that in my experience with this kit I found it difficult to make the turning movements with all four wheels assembled, so I decided to keep only two wheels in my robot and I added a swivel ball caster to the front and I positioned the battery at robot's back in order to balance weight. The robot was spinning very easily after doing this. Feel free to use other arrangements.

The Arduino board was positioned close to the motors, just ahead of it, with the USB connector pointing backwards. On top of Arduino was attached the motor driver shield.

The motor driver shield has a power jumper that indicates if there is a single DC power supply for both Arduino and motors. In this case it's better to use an external power supply for the motors, so the power jumper must be removed.

Since we are using in this project a power cell with USB connectors, I took a regular USB cable and I cut one end off, exposing its internal wires. The VCC and GND wires are usually colored in red and black respectively, but use a multimeter to make sure. These wires are connected to the external power header on the motor driver shield. The other USB wires (denoted D+ and D-) are not needed in this project.

The motor shield can drive up to four DC motors. They are labelled M1, M2, M3 and M4. In my project with 2 motorized wheels I wired the motors to the M3 and M4 connectors (two wires into each one). There are some configuration settings that can be adjusted later in another step in order to use other connectors.

The MFRC522 RFID module was positioned just ahead of Arduino on the acrylic plate. With this kit this leaves about 2cm space between the module's antenna and the floor, what should be enough for the purpose of detecting a RFID card on the floor.

Thus this is the final configuration of the lower acrylic plate: two DC motors mounted on the rear side, one swivel ball caster attached to the front side, a MFRC522 module attached close to it and an Arduino plus motor driver shield on the central part. Everything should be firmly attached to the acrylic plate using screws, spacers and hex nuts.

Step 3: Arduino Program

In this project the Arduino will be controlling the motor driver shield directly on demand of Raspberry Pi's logic. We use a serial communication between Raspberry and Arduino using the USB connection and a protocol specific for the purpose of this project.

This protocol basically defines some short commands (like 'f' to command it to move forward and 'b' to move backwards).

You may find the Arduino sketch files at this link:

[https://github.com/gustavohbf/robotoy/tree/master/arduino/RoboToy]

If you choose different motor driver numbers for your DC motors, you will have to change these variables in this program accordingly:

#define WHEELS  TWO_REAR

#define REAR_LEFT   4
#define REAR_RIGHT  3
#define FRONT_LEFT  1
#define FRONT_RIGHT 2

The first variable tells that we are going to control just two motors located behind the robot.

Each number on the following variables refers to the motor number used on the motor driver shield. So, out-of-the-box this sketch file tells that each robot will have just two rear motors and we are wiring the rear left motor with the driver number 4 (labelled M4 on the board) and the rear right motor with the driver number 3 (labelled M3 on the board). You may edit this file using any text editor of your choice or preferably using the Arduino IDE that you can find here: [https://www.arduino.cc/en/Main/Software]

Use the Arduino IDE to upload this sketch file into your Arduino using your computer and the USB cable connected to it.

You may want to follow the instructions in the following link in order to test your mount up to this point:

[http://github.com/gustavohbf/robotoy/wiki/Hardware-Assembly-Instructions#step-2-arduino-and-motor-driver-shield]

Step 4: Turret - Part 1

The turret comprehends the optical components of this robot. It includes the following parts:

1 RGB led
4 TSOP IR detectors (TSOP 4838)
1 IR led (TSAL 6200)
1 visible light led (e.g. yellow light with high brightness)
resistors and transistor
Raspberry camera

Before you proceed, check the datasheets for your components in order to identify the correct pin order. The TSOP 4838 I used here have the following pin order from left to right when the 'bump' is facing toward you: DATA, GND, VCC. The RGB led with common anode I'm using here have the following pin order from left to right when the anode (longer lead) is closer to the left side: RED, ANODE(+), GREEN, BLUE. For the remaining LEDs, just to remember, the longer lead is the positive (anode). If they have leads with the same length, look at the metal plate inside the LED. The smaller plate indicates the positive (anode) lead.

In order to hold for all these components together you can use this simple 3d turret model shared on this link:

[https://tinkercad.com/things/avhOTa6vCFk]

This project results in three 3d printed parts. Before gluing these parts together, connect the electronic components and join the wires. Do as follows:

Insert he RGB led's leads into the four small holes in the middle of the top part. You must keep them separated from each other.
Alongside the borders of this mounting part you will see four series of three dots markings. Carefully plug the four TSOP IR detectors on each one of them. The bump on each detector should be facing outwards.
Now it's time to fix cables to each of component leads. Choose different colors for each different purpose (e.g. black for GND, orange for VCC, yellow for data, gray for common RGB anode, red, green and blue for the corresponding RGB leads). If you have good soldering skills, try to solder one wire to each lead inside the turret. Otherwise, connect each lead to a dupont cable and use insulating tapes to stick each one of them. You may use a female-to-female dupont cable on the longer lead of RGB led (the common anode) and male-to-female cables for everything else. It's very important to use insulating tape to cover entirely all the leads because some of them will be touching the camera module. If you don't do this, you risk ruining your camera module.
Join two wires to the IR LED and another two to the visible light LED. Remember which wire color you choose for 'anode' and which one you choose for 'cathode' on each LED. Insert these LEDs through the turret's cannons.
The Raspberry camera is just about the right size to fit inside the turret 'head'. The square hole is the same size of the camera lens. You can hold it into the turret piece, leaving room for the wires of the cannon's LEDs just above it.
Glue the two parts forming the turret's 'head'.
Glue the turret's 'head' to its 'base'. Check the pictures above for the correct position.

Step 5: Turret - Part 2

The electronic circuit schematic is displayed above. You can either do this by soldering parts or by using dupont cables and a breadboard. In the pictures above I'm using a solderless design.

Basically you will have:

Three resistors with different values for the RGB led (one for each cathode). In this project I used 47 Ohms resistors for 'green' and 'blue' cathodes and 100 Ohms resistor for 'red' cathode. Check the datasheet for the LED you are using in order to choose proper resistance values. The current should be below 10 mA on each pin.
3.3V for the RGB led common anode (longer lead).
A 10 kOhm resistor connected to the transistor base.
Two 220 Ohms resistors connecting the transistor collector to the cannon's LED cathodes (they are connected in parallel).
5V for the cannon's LED anodes and for the TSOP detectors.
GND for the transistor emitter and for the TSOP detectors.

Note: the TSOP used in this circuitry are working at the 5 V level. In order to prevent it from damaging the GPIO of Raspberry, which should operate under 3.3 V level, we are going to use a voltage divider. According to the datasheet of TSOP 4838, they already have a 33 kOhm internal resistor between Vcc and 'data' pins. Using the internal 50 kOhm pull down resistors present in Raspberry Pi GPIO we would get the TSOP 'data' level down to a safe voltage level.

Step 6: Robot Upper Plate Assembly

On the upper acrylic plate we are going to attach the remaining parts:

Raspberry Pi on the front
Turret base on the central part. Breadboard with the turret circuitry next to it.
Battery (power cell) on the back

Start by fixing the turret base on the central part of the top acrylic plate. Use screws through the holes on the 3d printed base.

In case you didn't do it yet, format the microSD card in your computer and load it with the NOOBS installation manager. This project assumes you will be using the Raspbian operating system. In case you need help with NOOBS download and operation, check this link:

[https://www.raspberrypi.org/documentation/installation/noobs.md]

Once you have formatted and stored the NOOBS image in the microSD card, plug it in the Raspberry Pi.

Put the Raspberry Pi on the front of it with the USB connectors facing to the front. Use M2.5 screws to fix it to the plate.

The powerbank should stay behind in order to keep the overall weight balanced over the rear wheels. Fixing it on the acrylic plate may be tricky due to its shape. I was able to lock it in a steady position using the battery pack box. Use two angle brackets to fix the battery pack box on the acrylic plate and one more to support the battery inside it. I also had to cut small rectangular holes on the upper side of the pack box in order to expose the battery USB connectors.

The breadboard containing the turret circuitry was positioned to its side. The camera module flat cable was connected to the Raspberry CSI port (remember the Raspberry USB connectors should be facing ahead in order to prevent connecting the camera the wrong way). Then all connections to the Raspberry GPIO were made according to the diagram above. Overall there will be:

One 3.3 V power line going to the RGB led anode lead directly
One 5 V power line going to the breadboard which in turn will power all four TSOPs and the two cannon LEDs.
One GND line going to the breadboard which in turn will ground all four TSOPs and the transistor emitter lead used by the two cannon LEDs.
Three GPIO used as 'output' to the RGB led cathodes (red, green and blue through resistors on the breadboard)
Four GPIO used as 'input' from the four TSOP detectors.
One GPIO used as 'output' to the transistor base through a 10 kOhm resistor (used for turning on the cannon LEDs).
Seven pins to be connected to the RFID module attached on the lower acrylic plate.

Step 7: Final Connections

You are almost done with the hardware part of this project.

Attaches the six cylinder screw columns over the lower plate (were lies the Arduino board and all the motors). Use three on each side of the chassis.

Connect the Raspberry Pi on the upper plate to the MFRC522 RFID module on the lower plate using seven wires as shown on the table above.

Take a USB cable to connect to the Arduino board mounted on the lower plate and pass it over the small round hole on the upper plate.
Attaches the upper plate over the lower plate using six screws with the six separating cylinders. Plug the USB cable coming from the Arduino into one of the USB ports of Raspberry. Use the 'L' angled USB connector as needed.

Plug the USB cable connected to the 'external power' connection of the motor driver to the 2A USB port on the battery.
Plug the USB cable to the micro USB power port on Raspberry Pi and the other end to the other USB port on the battery.

Turn on the battery and hopefully everything will turn on nicely.

Step 8: Raspbian Setup

This project assumes you will be using Raspbian operating system in your Raspberry Pi.

There are some software parts needed by this project:

Raspbian operating system
Java JDK 1.8
UV4L + WebRTC
WiringPi + Pi4J
RXTX
SPI bus and camera module enabled

If you did not have time to prepare the Raspberry Pi for SSH session using WiFi, you will need to plug a monitor to the Raspberry Pi using its HDMI connector and also plug mouse and keyboard using its extra USB ports.

There are several good tutorials about setting these features. In case you need some references, I'll list some of them here:

Installing Raspbian: [https://www.raspberrypi.org/learning/noobs-install/]

Setting Wifi: [https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md]

Enabling SSH: [https://www.raspberrypi.org/documentation/remote-access/ssh/]

It's important to make the Raspberry accessible through WiFi because you won't like your monitor and peripherals hanging over your robot once it starts moving around. The SSH session will be used for configuration and maintenance, but you can replace it with a VNC session if you prefer a graphical interface. You may find more information about VNC configuration here:

Install VNC: [https://www.raspberrypi.org/documentation/remote-access/vnc/]

In order to install the camera module with UV4L, check the 'step 2' on the following page:

[https://github.com/gustavohbf/robotoy/wiki/Software-Installation#step-2-camera-and-video-streaming]

Also check the 'step 6' on the same page for instructions about installation of the serial port library:

[https://github.com/gustavohbf/robotoy/wiki/Software-Installation#step-6-install-serial-port-library]

Also check the 'step 7' in the same page for instructions about enabling the SPI bus to be used with the RFID module:

[https://github.com/gustavohbf/robotoy/wiki/Software-Installation#step-7-enable-spi-bus-for-rfid-module]

In case you want to use the robot as an 'access point', also check the optional steps 4 and 5 on the same page.

[https://github.com/gustavohbf/robotoy/wiki/Software-Installation#step-4-install-mdns-multicast-domain-name-system]

[https://github.com/gustavohbf/robotoy/wiki/Software-Installation#step-5-install-access-point-daemon]

Step 9: RoboToy Program

Finally you must install and configure the program that will control everything on the robot and will service the game pages for your controlling devices.

I already shared the whole application including an embedded application server and all the game logic. It's called 'RoboToy' and it's totally open source free software. The GitHub entry page is located at:

[https://github.com/gustavohbf/robotoy]

In case you don't have git installed yet, you can install it with the following command line in Raspbian:

sudo apt-get install git

Download RoboToy source code from GITHub.

git clone  https://github.com/gustavohbf/robotoy.git

Build the binaries from source.

cd robotoy
chmod +x gradlew
./gradlew build

Expand the compiled binary to the local system.

sudo tar -zxvf ./build/distributions/robotoy-1.0.tar.gz -C /usr/local

Change configurations as needed with your favorite text editor. There are several settings that can be changed. The default settings should work well if you followed this instructable exactly as described here. Look for comments in the configuration file. The default configuration file is located at:

/usr/local/robotoy/config.properties

Run the 'checkup.sh' shell script in order to verify if all the system components required are installed and configured properly.

/usr/local/robotoy/bin/checkup.sh

This script will check if your system satisfies all RoboToy requirements (e.g.: Java version, other packages, etc.). If any item is shown as 'FAIL', you should review the installation steps described in software installation page in RoboToy Wiki.

Test if everything is ok by running RoboToy script as root.

sudo /usr/local/robotoy/bin/RoboToy

There should not be anything like this (look for phrases starting with 'Exception in thread' expressions):

Exception in thread "main" java.lang.NullPointerException
    at org.guga.... etc

The RGB led should be turned on. If it's not, there might be a problem with the service or with the hardware wiring.
You should be able to connect to the robot using the following URL in any computer on the same network (change the '<ip address>' for the IP address assigned to Raspberry.

http://<ip address>

If everything is ok, you can press CTRL + C in order to close the running service and install it as a 'service' that will run every time after boot.

sudo /usr/local/robotoy/bin/robotoy_service.sh install

Then you can start it at once running:

sudo service robotoy start

Note that this time there is not going to be messages displayed on the console. Every console messages generated by RoboToy service should be redirected to temporary log files located at /tmp. The temporary log filenames will have this signature:

/tmp/robotoy.out    -> For standard output generated by the RoboToy service
/tmp/robotoy.err    -> For standard error generated by the RoboToy service

The majority of information should be output to '.err' files. If you restart RoboToy service, a new file is created and the previous one is renamed with a 'prev' suffix appended to its name.

If you need more help about installation procedure or customization, please check the RoboToy project pages at GitHub.

Have fun!